Integrating multiple knowledge sources for word hypotheses graph interpretation

We present an integrated approach for the interpretation of word hypotheses graphs (WHGs) using multiple knowledge sources. Commonly, di erent knowledge sources in speech understanding are applied sequentially. Typically, speech understanding systems, such as the Verbmobil speech-to-speech translation system, rst use a word recognizer to determine word hypotheses, only based on acoustic and language model (LM) information. The resulting word sequences or WHGs are then segmented according to syntactic and/or prosodic information. Finally, these segments are interpreted by a parser or a stochastic process. Thus, it is impossible to use the knowledge of the syntactic-prosodic process, the parser or any other subsequent process to nd the best word sequence. In our new approach we use acoustic, prosodic and LM information to determine the best word chain, to detect syntactic/prosodic/pragmatic phrase boundaries and to classify dialog acts in one integrated search procedure, based on a WHG or a word lattice.